Reinforcement Learning Method Research Articles

A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making

In the field of Sequential Decision Making (SDM), two paradigms have historically vied for supremacy: Automated Planning (AP) and Reinforcement Learning (RL). In the spirit of reconciliation, this paper reviews AP, RL and hybrid methods (e.g., novel learn to plan techniques) for solving Sequential Decision Processes (SDPs), focusing on their knowledge representation: symbolic, subsymbolic or a combination. Additionally, it also covers methods for learning the SDP structure. Finally, we compare the advantages and drawbacks of the existing methods and conclude that neurosymbolic AI poses a promising approach for SDM, since it combines AP and RL with a hybrid knowledge representation.

Visual Reinforcement Learning for Dynamic Object Detection

Abstract. Object detection is a widely studied task in computer vision. Current methods often focus on images captured from appropriate viewpoints. However, there is a large disparity between objects observed from different viewpoints in the real world. Dynamic Object Detection (DOD) method automatically adjusts the camera viewpoint in a visual scene to sequentially find optimal viewpoints. Currently, the DOD tasks are usually modeled as a sequential decision-making problem and solved using reinforcement learning methods. Existing approaches face challenges with sparse rewards and training instability. To tackle these issues, we proposed a single-step reward function and a lightweight network, respectively. The single-step reward function, which provides timely feedback, gives an efficient training process for DOD tasks. The lightweight network with few parameters can ensure the stability of the training process. To evaluate the effectiveness of our method, we developed a simulation dataset based on UE4, which consists of 1800 training images and 450 testing images. The dataset includes five object categories: vans, cars, trailers, box trucks and SUVs. Experiments demonstrate that our method outperforms SOTA object detectors on our simulation dataset. Specifically, the average precisions(APs) are improved from 89.1% to 96.0% when using the YOLOv8 object detector.

Integrated learning framework for multistep pick-place-arrange of arbitrarily shaped objects in a narrow crate

To maximize the space occupied by unknown objects in a narrow crate and handle uncertain situations, we propose a novel human-like system-level learning framework: sequential infiltration synthesis of combined learning (SISCL). This framework leverages the synergies between picking by grasping, three-dimensional modeling, preferable placement, and seamless arranging by pushing in a systematic perspective. Within SISCL, a closed-loop system organizes different designing methods including reinforcement learning, knowledge induction, and analytic methods. This integration trains the system to autonomously package unknown objects and accommodate for uncertainties in measurements or executions, combining efficiency and flexibility. Initially trained in simulations, the SISCL framework underwent comprehensive evaluation in the laboratory. Furthermore, an initial validation was conducted on an extraterrestrial test ground. The results demonstrate the adaptability of the proposed framework to different shapes, materials, and stiffness of objects, as well as the capacity for uncertainties, thereby considerably reducing the need for precision in each sub-procedure in a human-like manner, facilitating seamless operations and efficient utilization of limited space.

A homotopy-based reinforcement learning scheme to optimal control for Markov switched interconnected systems

ABSTRACT In this paper, a homotopy-based reinforcement learning optimal control method is developed for Markov switched interconnected systems with unknown system dynamics. By utilising the subsystem decomposition method and parallel learning control method, the solution of the game coupled algebraic Riccati equations with jumping parameters is approximated. To dispense with the requirement of initial stability, a homotopy-based policy iteration is introduced, which can place unstable poles into a stable plane. In this regard, a model-free reinforcement learning method is presented to design the optimal controller for Markov switched interconnected systems. Finally, the effectiveness of the proposed method is verified by a numerical example and a practical example.

Energy Aware Smart Car Parking System Based Internet of Spatial Things

With the advent of smart and green cities, the development of energy-efficient smart parking systems has attracted researchers to reduce environmental pollution in smart cities due to reducing traffic congestion as well as waste of time and fuel consumption. This article investigates how to integrate the Internet of Spatial Things (IoST) with workload balancing and image processing in fog computing to build an energy-efficient smart parking system. The suggested system applies Q-Learning, a reinforcement learning method, to achieve workload balancing and then deploys image processing to detect vacant parking slots. The proposed method’s main objective of load balancing is to reduce energy consumption. The evaluation of the proposed system is done by comparing two case studies, fog-based and cloud-based IoST implementations, in the iFogSim simulator for various scenarios and scales. Moreover, it evaluates and compares the energy consumption of various devices. Experimental findings show that the devised system in fog-based IoST greatly reduces energy consumption with improves parking space availability in smart parking in contrast to the cloud-based deployment of smart car parking.

Learning vision-based robotic manipulation tasks sequentially in offline reinforcement learning settings

Abstract With the rise of deep reinforcement learning (RL) methods, many complex robotic manipulation tasks are being solved. However, harnessing the full power of deep learning requires large datasets. Online RL does not suit itself readily into this paradigm due to costly and time-consuming agent-environment interaction. Therefore, many offline RL algorithms have recently been proposed to learn robotic tasks. But mainly, all such methods focus on a single-task or multitask learning, which requires retraining whenever we need to learn a new task. Continuously learning tasks without forgetting previous knowledge combined with the power of offline deep RL would allow us to scale the number of tasks by adding them one after another. This paper investigates the effectiveness of regularisation-based methods like synaptic intelligence for sequentially learning image-based robotic manipulation tasks in an offline-RL setup. We evaluate the performance of this combined framework against common challenges of sequential learning: catastrophic forgetting and forward knowledge transfer. We performed experiments with different task combinations to analyse the effect of task ordering. We also investigated the effect of the number of object configurations and the density of robot trajectories. We found that learning tasks sequentially helps in the retention of knowledge from previous tasks, thereby reducing the time required to learn a new task. Regularisation-based approaches for continuous learning, like the synaptic intelligence method, help mitigate catastrophic forgetting but have shown only limited transfer of knowledge from previous tasks.

Deep Reinforcement Learning for Pairs Trading: Evidence from China Black Series Futures

Pair trading is one of the main methods of statistical arbitrage, mainly by taking advantage of the temporary price anomalies between related financial products with long-term equilibrium relationships to obtain arbitrage opportunities. In this paper, based on the co-integration method for the selection of allotment, the deep reinforcement learning method is integrated. The deep reinforcement learning method establishes an autonomous learning model and develops paired trading rules under different cycles. At the same time, the deep neural network model is used to implement the learning and training algorithm. This paper uses the data of Shanghai Commodity Exchange and Dalian Commodity Exchange on black futures, and the period is from January 2, 2014 to December 31, 2021. Three different periods were set for five models, namely simple threshold method (ST), simple threshold method based on pairwise cointegration (CA-ST), simple threshold method based on tripartite Cointegration (CA-ST-ALL), deep reinforcement learning method (DRL), and Deep reinforcement learning method based on pairwise cointegration (CA-DRL), and economic indicators were compared. The final results show that, combining the results of the three time periods, the deep reinforcement learning method based on pairwise cointegration performs better regarding return and risk control, validating the feasibility of deep reinforcement learning in China's future market.

Deep deterministic policy gradient algorithm: A systematic review

Deep Reinforcement Learning (DRL) has gained significant adoption in diverse fields and applications, mainly due to its proficiency in resolving complicated decision-making problems in spaces with high-dimensional states and actions. Deep Deterministic Policy Gradient (DDPG) is a well-known DRL algorithm that adopts an actor-critic approach, synthesizing the advantages of value-based and policy-based reinforcement learning methods. The aim of this study is to provide a thorough examination of the latest developments, patterns, obstacles, and potential opportunities related to DDPG. A systematic search was conducted using relevant academic databases (Scopus, Web of Science, and ScienceDirect) to identify 85 relevant studies published in the last five years (2018-2023). We provide a comprehensive overview of the key concepts and components of DDPG, including its formulation, implementation, and training. Then, we highlight the various applications and domains of DDPG, including Autonomous Driving, Unmanned Aerial Vehicles, Resource Allocation, Communications and the Internet of Things, Robotics, and Finance. Additionally, we provide an in-depth comparison of DDPG with other DRL algorithms and traditional RL methods, highlighting its strengths and weaknesses. We believe that this review will be an essential resource for researchers, offering them valuable insights into the methods and techniques utilized in the field of DRL and DDPG.

Multi-agent reinforcement learning method for cutting parameters optimization based on simulation and experiment dual drive environment

Improving production efficiency while ensuring product surface quality is a constant focus of manufacturers. Cutting parameter optimization is an important technique for ensuring high-efficiency and high-quality production. In this paper, a novel method for cutting parameter optimization that integrates multi-agent reinforcement learning with a dual-drive virtual machining environment is proposed. First, a feature extraction, fusion and generation model for cutting simulation and experimental data is proposed to solve the problem of incomplete data acquisition in the production process. Second, a Markov decision model for optimizing cutting parameters is defined, and a virtual machining environment driven by both simulation and experimental data is constructed. Third, a novel multi-agent reinforcement learning method called Q-MIX-MATD3, in which a twin delay deep deterministic policy gradient, value function decomposition and a teacher model are combined, is proposed to explore the cutting parameter optimization policy by interacting with the virtual machining environment. Finally, the proposed method is verified on a commutator production line. Moreover, the results show that the accuracy of the virtual machining environment driven by both simulation and experiment increases by more than 5 %, response efficiency increases by 31 %, and Q-MIX-MATD3-based cutting parameter optimization method reduces time cost by 98 % and achieves the optimization effect of the classical optimization method.

Vibration Control with Reinforcement Learning Based on Multi-Reward Lightweight Networks

This paper proposes a reinforcement learning method using a deep residual shrinkage network based on multi-reward priority experience playback for high-frequency and high-dimensional continuous vibration control. Firstly, we keep the underlying equipment unchanged and construct a vibration system simulator using FIR filters to ensure the complete fidelity of the physical model. Then, by interacting with the simulator using our proposed algorithm, we identify the optimal control strategy, which is directly applied to real-world scenarios in the form of a neural network. A multi-reward mechanism is proposed to assist the lightweight network to find a near-optimal control strategy, and a priority experience playback mechanism is used to prioritize the data to accelerate the convergence speed of the neural network and improve the data utilization efficiency. At the same time, the deep residual shrinkage network is introduced to realize adaptive denoising and lightweightness of the neural network. The experimental results indicate that under narrowband white-noise excitation ranging from 0 to 100 Hz, the DDPG algorithm achieved a vibration reduction effect of 12.728 dB, while our algorithm achieved a vibration reduction effect of 20.240 dB. Meanwhile, the network parameters were reduced by more than 7.5 times.

Reinforcement learning-based SDN routing scheme empowered by causality detection and GNN

In recent years, with the rapid development of network applications and the increasing demand for high-quality network service, quality-of-service (QoS) routing has emerged as a critical network technology. The application of machine learning techniques, particularly reinforcement learning and graph neural network, has garnered significant attention in addressing this problem. However, existing reinforcement learning methods lack research on the causal impact of agent actions on the interactive environment, and graph neural network fail to effectively represent link features, which are pivotal for routing optimization. Therefore, this study quantifies the causal influence between the intelligent agent and the interactive environment based on causal inference techniques, aiming to guide the intelligent agent in improving the efficiency of exploring the action space. Simultaneously, graph neural network is employed to embed node and link features, and a reward function is designed that comprehensively considers network performance metrics and causality relevance. A centralized reinforcement learning method is proposed to effectively achieve QoS-aware routing in Software-Defined Networking (SDN). Finally, experiments are conducted in a network simulation environment, and metrics such as packet loss, delay, and throughput all outperform the baseline.

Spacecraft relative on-off control via reinforcement learning

The article investigates the task of spacecraft relative control using reactive actuators, the output of which has two states, “on” or “off”. For cases where the resolution of the thrusters does not provide an accurate approximation of linear control laws using a pulse-width thrust modulator, the possibility of applying reinforcement learning methods for direct finding of control laws that map the state vector and the on-off thruster commands has been investigated. To implement such an approach, a model of controlled relative motion of two satellites in the form of a Markov decision process was obtained. The intelligent agent is presented in the form of “actor” and “critic” neural networks, and the architecture of these modules is defined. It is proposed to use a cost function with variable weights of control actions, which allows for optimizing the number of thruster firings explicitly. To improve the control performance, it is proposed to use an extended input vector for the “actor” and “critic” neural networks of the intelligent agent, which, in addition to the state vector, also includes information about the control action on the previous control step and the control step number. To reduce the training time, the agent was pre-trained on the data obtained using conventional control algorithms. Numerical results demonstrate that the reinforcement learning methodology allows the agent to outperform the results provided by the linear controller with the pulse-width modulator in terms of control accuracy, response time, and number of thruster firings.

A Comprehensive Evaluation and Comparison of Enhanced Learning Methods

This paper provides a comprehensive evaluation and comparison of current reinforcement learning methods. By analyzing the strengths and weaknesses of the main methods, such as value function-based, strategy gradient-based, and value and strategy-based methods, the differences in their performances on standard problems and the applicable scenarios are explored. Meanwhile, other methods such as Monte Carlo Tree Search (MCTS) and evolutionary methods are also briefly introduced. Through the research and analysis in this paper, it provides reference and guidance for choosing appropriate reinforcement learning methods, and promotes the development and application of reinforcement learning techniques in practical applications.

Integrated operations strategies for shared and privately-owned autonomous vehicles: A deep reinforcement learning framework

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%).

Evaluation method with digital expert on the criticality of car-following scenarios for autonomous vehicles testing

Evaluation of autonomous vehicles is one of the major challenges before they can be released. Due to the advantages in efficiency, cost, and safety, scenario-based simulation methods have recently received great attention. Even so, as the complexity and uncertainty exist in the real driving environment, the scenarios that autonomous vehicles may encounter are infinite. Therefore, it is necessary to classify simulation scenarios according to their criticality. It contributes to accelerating the evaluation processes. This paper presents a novel criticality evaluation method, based on a proposed Digital Expert, for car-following autonomous driving. The Digital Expert acts as the evaluator to evaluate the criticality of scenarios depending on their driving performance. Driving performance refers to the achieved degree of driving intentions. Firstly, a Digital Expert is established as the evaluator for the criticality of the scenario using the inverse reinforcement learning method. Then, based on the fact that the intention of Digital Expert is to maximize its internal reward function, the reward function is used to evaluate driving performance. Finally, calculating the criticality of the car-following scenario according to the mapping relationship between driving performance and criticality. Using the driving data in the NGSIM data set, this paper generates two groups of simulated car-following scenarios and evaluates the criticalities of the two scenarios. The experimental results show that the proposed criticality evaluation method can reasonably evaluate the criticality of car-following scenarios.

A novel controllable bias reinforcement learning method for distributed automatic generation control integrated with large-scale electric vehicles

Due to the introduction of large-scale renewable energy resources and flexible load access, strong random disturbances will bring to the current integrated energy system. The traditional control methods fall short in adequately addressing the frequency regulation needs of distributed systems. From the perspective of distributed automatic generation control (AGC), an optimal coordinated control method is proposed based on reinforcement learning. The proposed method is developed to mitigate the estimation bias of the action-values via the minimum action-value from multiple action-value estimates. Further, the bias of action-value estimates can be adjusted from positive to negative to enhance the control accuracy. Then this method uses behaviour utility trace to achieve adaptive adjustment of the exploration rate, preventing repetitive and unbalanced exploration. Simulation experiments have been performed on the two-area LFC model integrated with electric vehicles and the multi-area integrated energy system model to verify the effectiveness of the proposed control method.

M 3 Rec: A Context-aware Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation

Reinforcement learning (RL) has shown great promise in optimizing long-term user interest in recommender systems. However, existing RL-based recommendation methods need a large number of interactions for each user to learn the recommendation policy. The challenge becomes more critical when recommending to new users who have a limited number of interactions. To that end, in this paper, we address the cold-start challenge in the RL-based recommender systems by proposing a novel context-aware offline meta-level model-based reinforcement learning approach for user adaptation. Our proposed approach learns to infer each user's preference with a user context variable that enables recommendation systems to better adapt to new users with limited contextual information. To improve adaptation efficiency, our approach learns to recover the user choice function and reward from limited contextual information through an inverse reinforcement learning method, which is used to assist the training of a meta-level recommendation agent. To avoid the need for online interaction, the proposed method is trained using historically collected offline data. Moreover, to tackle the challenge of offline policy training, we introduce a mutual information constraint between the user model and recommendation agent. Evaluation results show the superiority of our developed offline policy learning method when adapting to new users with limited contextual information. In addition, we provide a theoretical analysis of the recommendation performance bound.

Enhancing UAV Aerial Docking: A Hybrid Approach Combining Offline and Online Reinforcement Learning

In our study, we explore the task of performing docking maneuvers between two unmanned aerial vehicles (UAVs) using a combination of offline and online reinforcement learning (RL) methods. This task requires a UAV to accomplish external docking while maintaining stable flight control, representing two distinct types of objectives at the task execution level. Direct online RL training could lead to catastrophic forgetting, resulting in training failure. To overcome these challenges, we design a rule-based expert controller and accumulate an extensive dataset. Based on this, we concurrently design a series of rewards and train a guiding policy through offline RL. Then, we conduct comparative verification on different RL methods, ultimately selecting online RL to fine-tune the model trained offline. This strategy effectively combines the efficiency of offline RL with the exploratory capabilities of online RL. Our approach improves the success rate of the UAV’s aerial docking task, increasing it from 40% under the expert policy to 95%.

Reinforcement learning‐based composite suboptimal control for Markov jump singularly perturbed systems with unknown dynamics

In this article, a model‐free parallel reinforcement learning method is proposed to solve the suboptimal control problem for the Markov jump singularly perturbed systems. First, since fast and slow dynamics coexist in Markov jump singularly perturbed systems, it may lead to ill‐conditioned numerical problems during the controller design process. Therefore, the original system can be decomposed into independent subsystems at different time‐scales by employing the reduced order method. In addition, two model‐free parallel reinforcement learning algorithms are designed to obtain the optimal controllers for the fast and slow subsystems, respectively. Moreover, within the framework of reinforcement learning, the controllers of the Markov jump singularly perturbed systems can be acquired without the systems dynamics. Finally, a numerical example is introduced to prove the effectiveness of proposed algorithms.

Intelligent agent for hyperspectral image classification with noisy labels: a deep reinforcement learning framework

ABSTRACT Label accuracy plays an important role in supervised hyperspectral image classification problems. However, labelling can be prone to various types of noise, mainly including human errors and incorrect pseudo-label predictions. Learning from datasets with noisy labels is a critical challenge in practice. To address this issue, we first propose a deep reinforcement learning method named HyperRefine to tackle the label noise problem for hyperspectral image classification. The method utilizes an agent to learn the accuracy of each sample label and determine its suitability to be used in training the classifier model. The agent is trained using a reinforcement signal based on the reward obtained from a small validation set that reflects performance on the target task, where the validation set is generated from the training set by a momentum-based recruiter. Compared to complex hand-crafted methods of label noise cleaning, HyperRefine is easy to implement, yet highly effective. The robustness of the model to different types of label noises is significantly improved by further combining a novel label refinement approach. Experiments on artificially added label noise and pseudo-label noise hyperspectral datasets show that the proposed method can achieve promising performance.